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Introduction 

The diverse processes of peer review are familiar within the scientific community, where 
they serve a critical role in the allocation of such scarce resources as research funds and 
journal space, as well as in helping to produce reliable knowledge (Chubin and Hacked 
1990). In part because of the perceived success of peer review for funding and 
publication, reformers have sought to incorporate peer review into the policy making 
arena in efforts to improve the use of scientific knowledge, reduce the occurrence of legal 
challenges, and achieve other political goals (Jasanoff 1990). Many federal agencies, 
particularly regulatory agencies but also evaluative ones, use standing advisory 
commidees, ad hoc panels, and other peer review mechanisms to evaluate their programs, 
in-house research, the scientific literature, and regulatory proposals ( Smith 1992; Jasanoff 
1990). 

Recent efforts by Congress have sought to expand the domain of peer review in some 
ways. Bills introduced in the 106 th Congress include: S. 746, to peer review cost-benefit 
and risk analyses of major rules, among other purposes; H.R. 574, to peer review 
regulations supported by scientific data; and H.R. 2639, to peer review the data used in 
standards promulgated by the Occupational Safety and Health Administration. 1 But even 
in such agencies as the Environmental Protection Agency, where peer review has become 
relatively well established (EPA 1998; 1997; 1992), questions remains about the 
appropriate use of scientific knowledge and how peer review can enhance (or retard) the 
regulatory process (Powell 1999). 

This paper provides an introduction and overview to the use of regulatory peer review in 
the United States and a preliminary assessment of the proposed legislation that would 
further establish regulatory peer review. It is an early portion of a two-pronged research 
agenda that seeks to elaborate both the logical structure and the detailed procedures of 
regulatory peer review and other aspects of the use of scientific expertise in the policy 
process. 

In elaborating the logical structure of regulatory peer review, the project builds on 
previous work on the logical structure of science policy. Following the traditional 
distinction in the science policy literature between "policy for science" and "science in 
policy" (Brooks 1968), this earlier work applies a principal-agent perspective to focus 
exclusively on the delegation inherent in funding research (Braun 1993; Guston 1996; 
van der Meulen 1998, Caswill 1998). Applying the same principal-agent perspective to 
"science in policy," the agenda seeks to articulate a more sophisticated analytical 
framework for peer review (and other uses of expertise) that both scholars in several 
disciplines and practitioners in advice-producing and -consuming arenas can appreciate 
(Guston in preparation). 


1 The US Supreme Court, in its Daubert v. Merrell Dow , GE v. Joiner, and Kumho v. Carmichael 
decisions, has likewise sought to expand the purview of peer review as one measure of the admissability of 
expert testimony in the courtroom. 


Guston 


2 


PM3001241774 


Source: https://www.industrydocuments.ucsf.edu/docs/qndk0001 



APSA Panel 25-5 (39-6) 


Regulatory Peer Review 


In elaborating the detailed procedures of regulatory peer review, the project extends an 
approach to studying science and scientists by following them out of the laboratory 
(Latour 1987) and into arenas in which they are called upon to come to judgments in a 
more public and accountable way. It also extends an approach to studying science and 
scientists that emphasizes observing how, through their rhetoric as well as the 
establishment of policies and procedures, they attempt to demarcate their vision of 
scientific from non-scientific activities (Gieryn 1995; jasanoff 1990). Such detailed 
scrutiny, for example, can yield a more complete, scholarly understanding of consensus 
and consensus formation in regulatory science, as well as create some precise, policy¬ 
relevant knowledge about the decision making of expert advisory committees (Guston 
1999). Ultimately, the project will synthesize both perspectives in a way that has been 
done for "policy for science" (Guston 2000). 

This paper provides some preliminary descriptive work for the agenda. It begins by 
attempting to define peer review and making two sets of analytical distinctions — the first 
among among four different procedural aspects of peer review (in order to set the stage 
for detailed procedural research) and the second among three different types of peer 
review (in order to clearly identify the category of regulatory peer review). The next 
section reviews the literature that deals with regulatory peer review in particular. Next, 
the paper presents some brief case studies of how agencies already use peer review in 
regulatory and evaluative missions. Finally, the paper assesses whether the legislation 
the 106 th Congress has been considering is worthwhile and what additional research is 
necessary on regulatory peer review. 

Analytical Distinctions: Definitions, Aspects, and Types of Peer Review 

Defining Peer Review 

Traditional accounts locate the origins of peer review in the 17th century Royal Society, 
which implemented a practice of distributing manuscripts to its members for commentary 
and advice prior to publication in the Society's Philosophical Transactions 2 Former 
chemist and science policy scholar Arie Rip (1985:84) notes that "the Jesuit astronomers 
of the Collegium Romanum, who critically evaluated Galileo's claims, and the 
mathematicians who advised about the imprimatur of his books" could just as easily be 
cited as the original peer reviewers, but such an identification would be contrary to other 
scientific myths. 

Rip's point is that locating a particular origin of peer review is, in part, presuming to 
define its role in the scientific community in a particular and potentially uncritical way. 
While his point is well-taken, for the purposes of this paper the perspective of peer 
review developed by the General Accounting Office (GAO 1999) in its survey of federal 
practices seems sufficient: peer review is the process of reviewing of scientific or 
technical merit by competent persons with no unresolved conflicts of interest. GAO 
arrived at this definition empirically, through a survey of all federal research and 


2 Zuckerman and Merton (1971) focus on the Transactions in their study of the institutionali 2 ation of JPR; 
Kronick (1985:24) identifies a number of antecedent practices, however. 
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development (R&D) funding agencies. Key to the definition's usefulness, however, is 
further defining what constitutes scientific or technical merit, who the competent persons 
are and how they are selected, what conflicts of interest need to be resolved in what ways, 
and how the process of review relates to outcomes. 

Aspects of Peer Review 

In order to contextualize peer review beyond the ability of a definition like GAO's to do 
so, the paper examines peer review practices along four dimensions critical to its 
performance and management: 

1. Jurisdiction: What scope of scientific propositions or products are subject to review, 
who decides, and how? (This aspect relates to the constitution of scientific or 
technical merit.) 

2. Format: What is the format of the peer review? Does it use panels or individual 
reviewers? Are they masked reviews? What is the relationship between the peer 
reports and the decision made? (This aspect relates to the influence of process on 
outcome.) 

3. Instructions: What are the instructions given to reviewers? Do they specify certain 
criteria for particular consideration? (This aspect relates to both merit and process.) 

4. Participation: What rules govern participation in peer review? With what kind of 
discretion and under what restrictions do reviewers participate? (This aspect relates to 
both competent persons and conflicts of interest.) 

Clearly, the content of the dimensions of jurisdiction, format, instructions, and 
participation will vary over journal, funding, and regulatory peer review. Part of the 
purpose of the paper will be to elaborate the similarities and differences in this way. 


Types of Peer Review 

This paper addresses three primary types of peer review: funding peer review, journal 
peer review, and regulatory peer review. 3 Institutionally and, often, procedurally distinct, 
these three types of peer review share a roughly similar timing in their breadth of 
influence in the American polity. Although there had been isolated episodes of review 
early in the history of the Republic's scientific institutions, peer review as a system did 
not become institutionalized generally until the middle of the 20 :li Century. 

In funding peer review (FPR), program managers (or other officials) seek the opinions 
of experts about the worthiness of research proposals for financial support. Among the 
earliest examples of funding peer review are; the Smithsonian Institution, one of the 
nation’s oldest public scientific organizations, which created an advisory committee for 


1 It does not address other aspects of peer review, including: the evaluation of academic personnel for 
promotion and tenure; the selection of individuals for membership in honorific societies and other awards; 
die professional reviewing of patient records in hospitals (Garfield 1988a); and the evaluation of classroom 
teachers by other teachers. 
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reviewing and recommending funding proposals in the 1840s; the Navy Consulting 
Board, chaired by Thomas Edison, which in 1915 began reviewing requests for funding 
from inventors (Savage 1999); andthe Hygienic Laboratory, the predecessor to the 
National Institutes of Health (NIH), which pioneered peer review in 1902 with a 
congressionally mandated advisory committee (Smith 1990). The National Cancer Act of 
1937, and later the Public Health Service Act of 1944, legitimated the awarding of grants 
through an advisory council to researchers in universities and other institutions, and with 
NIH's creation of its Division of Research Grants and the original review groups, it 
established its nascent peer review system (Smith 1990). 4 At the National Science 
Foundation (NSF), however, FPR was only implicit at the start. An early guide to NSF's 
programs left the process of evaluating proposals unaddressed, although NSF's governing 
body, the National Science Board (NSB), understood that after staff evaluated proposals, 
they would send them to advisory bodies before presenting them to NSB for statutory 
approval (England 1982). 5 Other programs, such as the Office of Naval Research and the 
Defense Advanced Research Projects Agency, granted greater discretion to government 
personnel. 6 The decision to rely on managers at defense agencies was linked to the 
decision to rely on contracts rather than grants as a mechanism of funding (Sapolsky 
1990). 7 In the federal government, the strong manager alternative has generally thrived 
only in defense areas, where research has more explicit, consensual programmatic goals. 8 
Strong managers also thrive in private foundations that give research grants. Beginning 
in fiscal year 1996, the Office of Science and Technology Policy (OSTP) and the Office 
of Management and Budget (OMB) "have jointly provided annual direction to agencies 
encouraging them to emphasize the funding of peer-reviewed research over non-peer- 
reviewed research" (GAO 1999:4). Neither OSTP nor OMB provide agencies with 
concrete guidance on how to perform peer review, however. 9 

In journal peer review (JPR), the editors of professional journals seek the opinions of 
experts about the worthiness of manuscripts for publication. An early example of peer 
review for publication occurred when President Thomas Jefferson, who commissioned 
the Lewis and Clark expedition, requested the American Philosophical Society to review 


4 Sapolsky (1990:95n29) suggests that NIH found a model for peer review in the Rockefeller Fund, which 
began the practice in 1930 (but Rockefellers reasons for doing so are obscure). Sapolsky adds that NIH 
originally had a two-tiered process including academic scientists in one tier and government scientists in 
the other, but the role of the latter diminished as the framer objected. 

William E. Benson, assistant to the director of NSFs Earth Sciences program in tile 1950s, distinguished 
between the various meanings of peer review, opining "that NIH peer review is not just peer judgment but 
almost peer decision.,,, whereas here in NSF the system is peer advice" (England 1982:358). By this, 

Benson meant that program officers at NSF had greater discretion over awards than counterparts at Nil I. 

OTA (1991:127) suggests that manager discretion has become more explicit at both NSF and NIH. 

At ONR, however, managers might make informal inquiries of colleagues (Smith 1990; Sapolsky 1990). 

In this choice, however, ONR ignored the use of the National Academy as a reviewer for Army and Air 
Force projects (Sapolsky 1990:94-95). 

Manager discretion may have some advantages over peer review, including its ability to gauge project 
relevance, implement programmatic changes more rapidly, and fund proposals that may be counter to 
prevailing fashion (OTA 1991; 126), Some scientists speculated, however, that ONR managers favored 
their alma mateis and supported prominent scientists simply for die prestige of their portfolios (Sapolsky 
1990:96). 

OSTP has given guidance to agencies about the use of peer review for the purpose of program evaluation 
under the Government Performance and Results Act of 1993. See NSTC (1996). 
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the expedition's report prior to its publication (Savage 1999:5). Although some 
professional journals may have used informal varieties of peer review, JPR did not 
become widespread among scientific journals until the mid-20 th century, in part propelled 
by the increasing specialization of modem science (Chubin and Hackett 1990). It was 
not until the mid-1960s that major journals began publishing information about their peer 
review processes (Hargens, in CBE 1991). By that time, JPR was widespread but not 
universal: Zuckerman and Merton (1971) cite a 1968 study that found 71% of 156 
journals in 13 countries made some use of peer review, including 47 of 49 journals in the 
United States. More recently, Locke (1985) reports that more than three-quarters of 
scientific journals in the West use peer review, and that editorial review in China and the 
Soviet Union was not dissimilar. 

In regulatory peer review (RPR), a slightly more diffuse category, in which government 
officials seek the opinions of experts about the quality of scientific and technical inputs to 
regulatory or other policy decisions. Also in mid-century, federal agencies began to 
establish a peer review system by appointing scientific advisory committees to keep 
themselves abreast of technical advances, oversee their scientific programs, and 
legitimate their policy decisions (Jasanoff 1990; Smith 1992). More than any other 
Western democracy, the United States makes use of outside experts to advise its 
governing institutions (Weiss 1992), and the use of ad hoc advisory committees dates 
back to the early days of the Republic (Smith 1992). 10 During the Civil War, Congress 
created the National Academy of Sciences, not only as an honorific institution, but also to 
"investigate, examine, experiment, and report on any subject of science” when called 
upon by government (NAS 1985). 1 ' NAS created the National Research Council (NRC), 
its operating arm, in 1916, which in 1918 was perpetuated by a request from President 
Wilson. War again sparked a turn to technical advice, with the formation of the National 
Advisory Committee on Aeronautics in 1918. 

The real proliferation of advisory committees, both technical and otherwise, occurred 
with the management of the twin crises of the Depression and World War II. The 
Depression-era Science Advisory Board, for example, failed in its attempt to bring 
scientific thinking and technological applications to bear on the social problems of the 
1930s (Hart 1998:72-75). A chain of advisory boards relating to defense science, 
culminating in the creation of the Defense Science Board in 1956, has been an example 
of the "successful use of science advisers" (Smith 1992:48-67), 12 and the defense model 


I c While this greater use of outside experts is patently true for the executive branch of government, it may 
be less true far the legislative branch. See Bryner (1992) for a pessimistic vision of Congress's use of 
policy analysis. See Vig and Paschen (1990) for a review of relatively robust parliamentary offices of 
technology assessment in Europe in the years since the demise of the U.S. congressional Office of 
Technology Assessment, See Bimber (1996) for OTA's histoty. 

II The National Academy complex evolved over time with continuing federal involvement The National 
Academy of Engineering and the Institute of Medicine were established under the NAS charter in 1964 and 
1970, respectively. See NAS (1985). 

12 For insight into the early politics of this chain, see Blanpied (1995). 
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was very influential in the creation of the President's Science Advisory Committee later 
in the second Eisenhower administration. i3 


The 1973 creation of EPA’s Science Advisory Board formally brought the advisory 
committee model to regulatory science (Jasanoff 1990; Smith 1992). Other agencies 
appointed advisory committees and increased their use of professional and honorific 
societies, including the National Academy of Sciences and others, for similar purposes. 
Because of the diversity of policies and problems that emerged at many agencies, the 
National Research Council (1983) recommended the heuristic separation (but not the 
radical institutional separation) of risk assessment and risk management in risk policy. 
With the assistance of OSTP, EPA implemented these changes and withstood additional 
external challenges (Jasanoff 1990). 

The proliferation of advisory committees in the twenty-five years after the war led 
Congress to pass the Federal Advisory Committee Act (FACA) in 1972, setting down 
rules governing their use and mandating some degree of openness in their activities and 
balance in their membership. 14 The General Services Administration (GSA), which is 
charged by the President to fulfill FACA's reporting requirements, estimates the number 
of FACA committees at just under 1000, a number that has been quite steady and a 
modest reduction in line with President Clinton's 1993 Executive Order 12838 (GSA 
1997). In fiscal year 1998, GSA counted 23% of FACA committees as 
scientific/technical and another 14% as grant review (GSA 1999), percentages consistent 
with previous years. In 1997, a Federal court ruled that the government "utilized" the 
National Academy complex under FACA and since, with some legislative direction (cite 
law), the Academy has altered its procedures to provide greater openness (GAO 1998). 

In recent years, particularly since the 104 th Congress, ideas about regulatory reform, tort 
reform, and other issues in the application of science to decision making have included a 
role for regulatory peer review. 


Peer Review in the Regulatory Environment 

There is no coherent literature on RPR and only a few major works of scholarship and/or 
analysis that bear directly on the topic. Earlier work, such as Ashford (1984), is still very 
much in the tradition of participants in the process offering observations on and, perhaps, 
analytical frames for their experiences. Perhaps the most significant, broad scholarly 
treatment of RPR is Jasanoff (1990), although Smith (1992) is an important and relevant 
work as well. Other works deal more generally with science and risk in the regulatory 


' J PSAC persisted through the second Nixon administration, when President Nixon disbanded it Congress 
reconstituted a presidential science apparatus in 1976, however, but it excluded an advisory council like 
PSAC. A White House Science Council was created in the Reagan administration, and a new President’s 
Council of Advisers on Science and Technology was created in the Bush administration and endures 
through the Clinton administration. See Golden (1988) for various short takes on presidential science 
advising. 

14 Smith (1992:24) describes how the congressional impetus for FACA was driven by a perceived abuse of 
secret deliberations by advisory committees in USDA and EPA in their consideration of 2,4,5-T (dioxin). 
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process (Graham and Weiner 1995; Carnegie Commission 1993a; Breyer 1993; Graham, 
Green, and Roberts 1988). There are also some studies that bear on RPR at particular 
agencies, notably Powell's (1999) recent study of the use of science at EPA (a discussion 
of which is deferred to the section on EPA). Together, the literature presents a 
patchwork perspective that suggests that the use of science in regulatory decision making 
has improved over time but still could use additional refinement, particularly with regard 
to its FACA responsibilities. 

Ashford (1984) compares and contrasts the experience with science advisory committees 
at EPA and OSHA, two agencies he had the opportunity to serve. He focuses mostly on 
the intended functions of these advisory committees, making some helpful observations 
and introducing some useful analytical categories along the way. For example, he 
distinguishes among permanent (statutory), quasi-permanent (standing but 
administratively created), and ad hoc committees. He then claims that ad hoc-ness at 
OSHA led to an uneven approach to standard-setting, and that EPA's technique of 
drawing ad hoc committees from its SAB membership permits both the flexibility that ad 
hoc-ness seeks and the continuity that OSHA does not achieve. 

Ashford also places the burden of performance significantly on the FACA-mandated 
balance of advisory committees, suggesting that FACA’s language and legislative history 
seek balance in order to: increase the role of those affected by decisions in advising; 
avoid over-representation of any particular geography, interest, or discipline; include 
consumers and public interest representatives; include informed laypersons; and prevent 
dominance of any private interests from making special claims on decision makers. He 
summarizes these in three dimensions of balance: competence, discipline, and allegiance. 
The scientific community - which (through the voices of NAS and AAAS) supported 
President Reagan's veto of an explicit balancing law even in the wake of his SAB "hit 
list" — may be a barrier to balance because its "desire to maintain the illusion of neutrality 
of science and scientists...may, as a practical matter, prefer covert stacking to overt 
balancing" (Ashford 1984:79). Balance does not prevent misuse, but it may avoid misuse 
under the perception of fair use. 

Jasanoff(1990) also does not study RPR directly. Rather, she is interested in the broader 
question of how the role of science in policy can be restrained, useful, legitimate, and 
relevant all at the same time. Jasanoff begins her analysis by identifying a pattern in 
technical decisions from 1975 to 1980 in which proposals to regulate potential hazards, 
based on studies or risk assessments performed in-house, were followed by challenges 
from external experts and resulted in the withdrawal of regulatory proposals and 
subsequent damage to the reputation of agencies. She concludes, with other observers, 
that these cases (including nitrites, 2,4,5-T, Love Canal, and occupational cancer) 
suggested the need for more complete peer review within agencies; but she also argues 
that the particular resolution of these cases did not resolve the larger question of who 

would be the final arbiter of the technical competence of assessments: the agency, the 
courts, or an independent scientific body. 
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One response to these controversies sought to separate science from policy (either in 
separate institutions or, less radically, by establishing firmer barriers within the same 
institution). NAS ultimately rejected the institutional separation of risk assessment and 
risk management, for practical as well as intellectual reasons, instead advocating the 
strengthening of peer review as an alternative. But peer review, Jasanoff argues, is a 
system originally designed for use in the realm of research science, which she 
distinguishes — by virtue of goals, methods, endpoints, and institutional context -- from 
regulatory science. 15 Also recognizing shortcomings in journal and funding peer review, 
including that peer review is not a fail-safe mechanism for determining integrity or 
quality and that it can open the window to questions of validity rather than necessarily 
resolve them, JasanofF is skeptical about the ability of peer review alone to solve the 
problem of scientific inputs to regulatory decisions. 

Jasanoffs case studies explore these questions of peer review and regulatory decision 
making: 

• In her case on EPA’s Science Advisory Board, she traces an evolution from 
politicization and confrontation in the early 1980s to a more cooperative relationship 
later in the decade, concluding that successful RPR by SAB is a complex result of the 
historical accident of a backlash to political mismanagement, the intentional strategy 
of interest representation, and the ability of SAB to seem aloof from policy decisions 
while subtlety blending policy and science. 

• The case of EPA's Clean Air Scientific Advisory Committee also shows an evolution 
— punctuated by congressional action to clarify CASAC's role — from controversy to 
accommodation, although EPA and CASAC did continue to dispute such science 
policy questions as what constitutes an adverse health effect, who were appropriate 
test subjects, and what constituted an appropriate risk assessment methodology. 

• EPA's Science Advisory Panel illustrates a more enduring adversarial relationship in 
the provision of advice for pesticide regulation, especially over science policy 
questions about the nature of animal models of carcinogenic risk, the relative 
importance of individual studies versus the entire body of evidence, and questions 
about possible bias toward industry. 

• Jasanoffs cases of FDA's use of advisory committees show many of the same 
difficulties of science policy issues, the benefits of give-and-take between the agency 
and advisory committees, and the possibility that science advice might push an 
agency into an area contrary to settled legal interpretations. 

Jasanoff concludes that the institutional representation of science is important to making 
good regulatory decisions, but that negotiated rather than adversarial procedures are 
superior. This institutionalized science in the form of peer review, however, needs to be 
balanced in its representation of multiple disciplines and interests and its familiarity with 
the particular challenges of regulatory (as opposed to research) science. Such review 
should occur early to prevent unsubstantiated progress, but advice should still leave room 
for discretion on the agency's part. 


15 Jasanoff (1995) also focuses on the distinctive category of regulatory science. 
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Smith's (1992) treatment of advisory committees is somewhat less nuanced but more 
broadly structured than Jasanoffs; it pertains less to RPR as such and more to the general 
problem of technical advising of executive agencies. After reviewing the history of such 
committees from the early days of the Republic and dwelling in some detail on the 
governing arrangements that encouraged and regulated advisory committees — 
particularly the Federal Advisory Committee Act, the Freedom of Information Act, and 
their antecedents — Smith also presents a series of loosely comparative case studies: 

• Smith’s study of the Defense Science Board demonstrates the successful long-term 
development of a tightly integrated advisory board, one that can survive particular 
controversies (such as the Strategic Defense Initiative) and challenges (such as a 
narrow base in the defense industry), but that may not be able to withstand the 
epochal changes in the role of defense and defense industries after the Cold War. 

• Smith’s portrait of EPA's Science Advisory Board shows remarkable likeness to 
Jasanoffs, as he traces the evolution of SAB from its early and self-serving vision of 
the simple provision of science advice to its latter and more nuanced development of 
expertise in regulatory science (which Smith (1992:69) defines as "a unique mixture 
of expertise and policy relevance"). 

• DOE Advisory Boards and NASA's Advisory Council are much less successful than 
DSB or SAB, largely because they operate in a more fragmented policy arena where 
the agency lacks basic consensus around its mission and political support is missing 
from top levels of government. DOE's boards operate without any consensus on the 
government role in civilian energy development and production, and NASA's 
Advisory Council has performed mostly as a "support group" for an agency embattled 
by fiscal difficulties, the sequelae of the Challenger disaster, and rifts in the scientific 
community about the nature of space exploration. 

• Science advice in the State Department is the least successful example, but perhaps 
still an instructive one, as it was not just that the State Department failed to commit 
serious resources and attention to science advice over time, but it neglected a standing 
advisory committee largely because of the preference of officials for informal rather 
than formal technical advice. 16 

• Smith also survey's the history of science advice to the President, finding that the 
President needs science advice, but that the utility of that advice will depend almost 
entirely on the President, so the science advisor needs to perform like any other 
policy advisor and not like a scientist, per se. 17 

From these examples, Smith concludes that science advisory committees need: a clear 
mandate relating to an important problem (for the agency or the nation); an identifiable 
client to receive their advice; an active chairperson with, if possible, good personal 
relations with the client; and balance among outlook but congruence of purpose. Like 
Jasanoff, he finds that science advisors need to do more than science to be successful, 
not just for themselves but for the nation: "the democratic dialogue is not well served by 


16 The State Department has recently made some changes to improve its access to scientific and technical 
expertise. 

17 The Office of Science and Technology Policy, directed by the special assistant to Hie President for 
science and technology (the President’s "science advisor”) has, through the National Science and 
Technology Council, emphasized managing the RAD enterprise and de-emphasized expert advice. 
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the simplistic notion that the advisory relationship is based just on objective information" 
(Smith 1992:65). 

In 1993, the Carnegie Commission on Science, Technology, and Government released its 
report on Risk and the Environment: Improving Regulatory Decision Making. Although 
the "examination of environmental and risk-related policymaking focuses on the role of 
science" and "stress[es] the importance of sound science" (Carnegie Commission 
1993a: 117, 118) regulatory peer review does not loom large in the Commission's 
assessment. In its recommendations regarding "science, risk, and regulatory decision 
making," the Commission suggested that "[rjegulatory agencies should critically evaluate 
and take deliberative steps to improve their internal scientific capabilities and their means 
of integrating scientific and technological considerations into agency decision-making 
processes" (21). The Commission recommended that, to achieve this improvement, 
agencies should "seek advice from other government agencies" and appoint "[individuals 
with both public policy and scientific expertise., .more frequently to senior positions" 

(21). Agencies should also "enhance their long-range planning capabilities by 
strengthening the linkages between research and regulatory policymaking efforts," 
including intramural research and planning and extramural research (22-23). Although 
the Commission acknowledged that "[ejxtemal science advisory boards serve a critically 
important function in providing regulatory agencies with expert advice on a wide range 
of issues [and in] serving] as a bridge between government, industry, and academia," it 
also cautioned that "federal agencies should not rely so heavily on external advisory 
boards that internal scientific capabilities are neglected or are automatically assumed to 
be less credible than those undertaken by nongovernmental experts" (90-91). 

Thus, peer review for regulations was not explicit in the Commission's approach to 
improving regulatory policy, but it was important in its approach to maintaining the 
quality of inputs to regulatory decisions. For example, the Commission promoted a 
comparative risk approach for regulation, and it argued that "[tjhe sine qua non of 
relative risk analysis is scientific data" (82). It wrote, 

Utility and reliability are important criteria for inclusion of data in any 
inventory, and chief among the indicia of reliability of scientific data 
would mostly likely be favorable peer review. Data published in peer- 
reviewed journals would a priori be acceptable for incorporation. The 
agencies could also establish science advisory groups to help review 
potentially important but unpublished data....[But r]egulatory decisions 
often cannot wait until journal-quality science is available, and decision 
makers in such situations need access to any existing data that are not 
patently unreliable (85). 


Short Cases in Peer Review: Regulatory Agencies 

Many Federal agencies already practice forms of peer review for the science underlying 
their regulatory efforts. Other agencies with evaluative or assessment missions use 
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advisory committees or similar forms of review as well. This section reports on the RPR 
practices of a number of these agencies, in an agency-by-agency format. It begins with 
regulatory agencies (EPA, FDA, and OSHA), continues with other evaluative agencies 
(NTP, OMAR, NMFS,...) and the National Academy complex, and concludes with RPR 
activities in states. An assessment of RPR according the previously detailed categories of 
jurisdiction, format, instructions, and participation occurs in the synthesis and discussion 
section. 

Environmental Protection Agency 

EPA is an independent agency charged with regulating the quality of the environment 
under a variety of laws protecting air, water, and soil and other legislative mandates. 
Because of its politically vulnerable position at the intersection of vital political and 
economic interests and uncertain science, EPA has developed perhaps the most elaborate 
structures, policies, and procedures for RPR. This short case focuses on developments 
since Jasanoff (1990) and Smith (1992) examined EPA. 

Beginning in the early 1990s, EPA has focused extensive attention on its use of science in 
its own decision making. Safeguarding the Future: Credible Science, Credible Decisions 
(EPA 1992) — itself a product of a peer advisory panel — emphasized the importance of 
external peer review for EPA scientific and technical products, contact between EPA and 
external scientists, and the use of the best science in EPA decisions. Implementation of 
this agenda has taken some time, in part because of difficulties in developing standard 
operating procedures among various EPA divisions (Browner 1994). To this end. 
Administrator Browner created a Science Policy Council to "expand and improve peer 
review in all EPA offices." One of SPC's early products was "Guidance for Conducting 
External Peer Review of Environmental Regulatory Models" (Sussman 1994). 18 

A 1997 memo outlined the new policy for "peer review and peer involvement" in EPA 
(1997). The memo described peer review as a "particularly important type of peer 
involvement" that "not only enriches the quality of work products but also adds a degree 
of credibility that cannot be achieved in any other way. Further, peer review early in the 
development of work products in some cases may conserve fiiture resources by steering 
the development along the most efficacious course." The policy articulated that "major 
work products that are primarily scientific and technical in nature and may contribute to 
the basis for policy or regulatory decisions" normally should be peer-reviewed. Although 
the memo maintained that the policy "does not establish or affect legal rights or 
obligations," that "[mjinor variations in its application...are not a legitimate basis for 
delaying or complicating action," and that per review is not a formal part of nor a 


Powell (1999) reports that a 1994 Senate directive instructed EPA to seek help from NSF in the practice 
of peer review. Powell suggests, however, that NSF may not be a good model because it can afford a sole 
emphasis on proposal quality; whereas, EPA must consider relevance and multi-disciplinary problem 
orientation. 

19 A more complete chronology exists in Powell (1999), which also describes a 1996 GAO report that 
found uneven progress on implementing peer review and an internal study conducted by EPA’s Office of 
Research and Development that repotted few deviations from peer review requirements except in the area 
of models. 
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substitute for rulemaking, the memo does recommend that the remarks of peer reviewers 
should be part of a decision record if the relevant work products have been reviewed. 
The memo also emphasized the need for careful management of a peer review process 
that could be "time-consuming and expensive" and is sensitive to issues of credibility 
should conflicts of interest occur. 

Shortly thereafter, the Peer Review Advisory Group, a subcommittee of the SPC, 
published the Peer Review Handbook (EPA 1998). The Handbook serves as a guide for 
EPA staff and managers on the "organization and conduct of peer review." It recognizes 
a diversity of peer reviews "depending on the nature of the work product, relevant 
statutory requirements, and office-specific policies and practices (p. 9) and states that the 
determination of rules as "major" and therefore appropriate for peer review "will largely 
be case-by-case" and require the judgment of an assistant administrator, regional 
administrator, or their designee (p. 26). Criteria for peer review status include work 
products that: establish a significant precedent, model, or methodology; address 
controversial or emergent issues; have cross- or inter-agency implications; are costly; 
apply innovative approaches to problems, processes, or methodologies; and satisfy 
statutory or other legal mandate for peer review. The Handbook also exempts from 
review "scientific and/or technical methodologies or information [that] are commonly 
accepted in the field of expertise," although such exemption still would require 
documentation of acceptance in the field (p. 30). It appears that EPA does not regularly 
review cost-benefit analysis, and the agency may understand CBA as being such a 
commonly accepted method as not requiring review. 


A Peer Review Leader (PRL) manages the review, and a Peer Review Coordinator 
oversees it. The PRL is supposed to create a formal record of each review and its 
outcome, sufficient for "an uninvolved person to understand what actually happened and 
why" (p. 37). The Handbook accepts the use of internal peer reviewers in some 
circumstances, provided that they are not employed in the same unit that produced the 
work. It also describes a variety of other acceptable peer reviews, including ad hoc and 
standing committees and workshops, and it suggests that when an ad hoc review occurs, 
more than one reviewer is usually necessary, although two or three may be acceptable. 

The Handbook recommends that informal review of the composition of reviewers may be 

helpful and that experts' positions should be balanced along the continuum of "technically 
legitimate points of view" but not be "too far to either extreme" (p. 47). The PRL 
formulates the charge to these reviewers, and the Handbook encourages attention to the 
charge by stating that, "[a]s a general mle, the time drafting a good charge letter is well- 
spent and is necessary for an effective peer review" (p. 43). Such review is supposed to 
occur prior to public comment. 

The Handbook recognizes a relatively wide definition of conflict of interest (or 
perceived conflict of interest) as "when a person is affected by his/her private interests, or 
when he/she or his/her associates would derive benefit from incorporation of their point 
of view in an Agency product" (p. 47). It points to established forms and mechanisms for 
managing conflicts or potential conflicts at standing advisory committees. The 
Handbook acknowledges that interested experts may be among the most knowledgeable 
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and allows for their participation under the conditions of disclosure of (potential) 
conflicts and balance among the reviewers (although in some conditions, e.g., for a 
potential reviewer with "a client or employer with a direct financial stake in the matter" 
(p. 48), the conflict will be "so direct and substantial" as to disqualify the expert). The 
Handbook equivocates on the applicability of the Federal Advisory Committee Act to 
peer review, suggesting that in some cases review by individuals (rather than consensus 
committees) or by committees of contractors may not be constrained by FACA. 

Food and Drug Administration 

FDA is an agency of the Public Health Service, and its mission is to ensure the safety of 
processed foods and the safety and efficacy of drugs and medical devices, among other 
related products. 

Central to its function of approving new drags and devices is FDA's review of the data 
submitted along with new drug applications (NDAs), and the principal activity of the 
advisory committees at FDA’s Center for Drug Evaluation and Research is to make sure 
that these NDAs comply with statutory requirements for "substantial evidence" 

"consisting of adequate and well-controlled investigations," CDER makes use of these 
committees "so that final agency decisions will have the benefit of wider national expert 
input. Committee recommendations are not binding on CDER, but the agency considers 
them carefully when deciding drug issues." 20 JasanofF(1990:154) reports that, as of 
1986, FDA had "thirty-nine standing committees and panels associated with the 
Commissioner's Office, the National Center for Toxicological Research, and the Centers 
for Drugs and Biologies, Devices and Radiological Health, and Veterinary Medicine." 21 
In stark contrast to the practice of journal editors and reviewers, however, FDA does not 
assume that the clinical inquiries described in the NDA have occurred. Rather, FDA 
engages in both random and for-cause audits of the clinical research unpinning NDAs 
(Nightengale and Kimbrough 1989). 

CDER also takes advantage of non-FACA committees, for example, the Statistical Policy 
Coordinating Committee, which develops policies pertaining to the design and evaluation 
of industry studies and reviews CDER programs, staff, and publications with the goals of 
maintaining good statistical practice and creating a consistent approach to statistical 
analysis (FDA 1998). There is also a Biopharmaceutics Coordinating Committee, a 
Chemistry, Manufacturing, and Controls Coordinating Committee, and a 
Pharmacology/Toxicology Coordinating Committee, as well as several management 
coordinating committees for projects and information technology. The Carcinogenicity 
Assessment Committee is yet another internal committee responsible for providing 
CDER with additional review of carcinogenicity studies and support for other issues in 
carcinogenicity. 


20 See http^/www.fdagov/cder/handbook/index.htm 

21 JasanofF (1990) reports that many of these panels had been appointed by the FDA commissioner or 
division, but then die Department- level began exercising more control over appointments — not necessary a 
salutary development as physicians were not above politicking for appointments. 
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FDA has incorporated peer review as an important part of its centers' missions, as 
expressed in strategic planning documents. For example, the National Center for 
Toxicological Research conducts fundamental and applied research on the mechanisms of 
toxicity, providing analytical skills and experience that supports the agency's regulatory 
function. 2 It maintains a Science Advisory Board, composed of scientists from industry, 
academia, and consumer groups, and supplemented by topic-area experts from all sectors 
that FDA regulates. NCTR relies on SAB to review its programs. Studies conducted by 
NCTR researchers are verified by internal and external peer reviews, and NCTR 
encourages researchers to publish in peer-reviewed journals. 23 

FDA also uses peer review as an indicator or criterion for inputs into some decision 
making. For example, the agency recently issued a final rule on the dissemination by 
manufacturers of information about off-label uses for drugs, devices and biologies (FDA 
1998:64555), requiring that any such information disseminated "must be in the form of 
an unabridged reprint or copy of a peer-reviewed journal article or reference publication." 

Occupational Safety and Health Administration 

OSHA is an agency of the Department of Labor. OSHA has two primary mechanisms for 
receiving external advice: statutory advisory committees and ad hoc committees. 

The Occupational Safety and Health Act of 1970 instituted a National Advisory Council 
on Occupational Safely and Health (NACOSH). Its puipose is to "advise, consult with 
and make recommendations" the Secretaries of Labor and Health "on matters relating to 
the Administration of the Act." The Council has a corporate membership, with two 
members each representing management, labor, occupational health professions, and 
occupational safety professions, and four represent the public. All are "selected upon the 
basis of their experience and competence in the field of occupational safety and health." 
Members serve two-year terms, and the Council holds between two and five meetings per 
year. The Council provides advice and recommendations upon the approval of a majority 
of its members, and all concurring or dissenting opinions (as well as abstentions and 
absences) must be recorded; individual members may also submit their own views on 
matters considered by the Council. The Council can also request the assistance of experts 
who can, at its discretion, participate in its deliberations. The Council also adheres to the 
rules of federal advisory committees. 

The Contract Work Hours and Safety Standards Act also created an Advisory Committee 
on Construction Safety and Health. The Assistant Secretary seeks the advice of the 
Committee by presenting it with a proposal, as well as "any pertinent factual information 
available... including the results of research, demonstrations, and experiments." Such 
advice must be sought in the formulation of a rule to promulgate, modify, or revoke a 
construction standard under the Act. The Council must submit its recommendations by 
the date the Assistant Secretary has designated, which must be no longer than 270 days 
from the initial consultation. The Council's recommendations are advisory to the 


22 See httpy/w\vw.fda.gov/one/Fy'00plan/NCTR00.htm ard hFtp:/Av\v , \v.fda.gov/or>e/fv01plan/nctif)l .htmi . 

23 In internal review, for example, statistical protocols are reviewed by members of the Biometry Staff 
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Assistant Secretary and the Secretary, and take place prior to the Federal Register 
process. 

The Occupational Safety and Health Act also allows the Assistant Secretary to appoint an 
ad hoc advisory committees, composed of no more than fifteen "persons qualified by 
experience and affiliation to present the viewpoint of the employers involved" and 
"persons similarly qualified to present the viewpoint of the workers involved 1 ' (Ashford 
1984: 74). 

Ashford (1984: 75) asserts that NACOSH has been "a major forum for discussing and 
clarifying controversial issues before unnecessary adversarial interactions occur," 
including such episodes as cancer policy, reproductive hazards, the science-law interface, 
and internal research needs and opportunities. He also points out that the ad hoc 
committees have been less central to OSHA standard-setting, and indeed that they have 
been appointed less often than not. There are cases of OSHA following the advice of ad 
hoc committees, e.g., coke oven emissions standards, and cases of it ignoring advice, e.g., 
pesticide exposures. Ashford (1984:75-76) points to the transitory nature of such 
committees as an important element to their lack of success, although the flexibility of ad 
hoc committees for particular regulatory tasks may be desirable. 

Short Cases in Peer Review: Evaluative and Assessment Agencies 

Worth distinguishing because regulatory agencies bear burdens (APA) that these do not. 


National Toxicology Program 24 

The National Toxicology Program (NTP) is an agency of the Department of Health and 
Human Services (DHHS), created in 1978 with the mission "to evaluate agents of public 
health concern by developing and applying the tools of modem toxicology and molecular 
biology" (NTP 1999, 2). NTP is a collaboration among the National Institute of 
Environmental Health Sciences of the National Institutes of Health (NIEHS/NIH), the 
National Institute for Occupational Health and Safety of the Centers for Disease Control 
and Prevention (NIOSH/CDC), and the National Center for Toxicological Research of 
the Food and Drug Administration (NCTR/FDA). 

NTP uses a Board of Scientific Counselors (BSC) to provide peer review for a number of 
agency activities. The full BSC provides oversight for the Center for the Evaluation of 
Risks to Human Reproduction. The Center organizes ad hoc Scientific Review Panels, 
which review background documents and evaluate two or three chemical each year for 
evidence of their possible adverse effects on fertility and childbearing; the resulting 
reports are published in Environmental Health Perspectives. The Scientific Review' 
Panel draws its members from an Expert Registry comprised of scientists nominated by 
the public. 


24 Most information in this section drawn from NTP (1999). 
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The BSC's Technical Reports Review Subcommittee reviews toxicological studies of 
substances that have been nominated for study from a variety of sources and approved by 
a public process that includes review by both the full BSC and an internal Executive 
Committee. The BSC's Report on Carcinogens Subcommittee reviews the record on 
substances that have been nominated as "known" or "reasonably anticipated to be" human 
carcinogens in preparation for their possible listing in the biennial Report on 
Carcinogens 25 NTP staff recommend potential members of BSC to the NTP director, 
based on the Board's needs for technical expertise and balance; the director nominates 
members, who are appointed by the Secretary, DHHS. 

NTP also maintains a Center for the Evaluation of Alternative Toxicological Methods 
(NICEATM), which provides operational support to the Interagency Coordinating 
Committee on the Validation of Alternative Methods (ICCVAM), established in 1997 to 
"reduce, refine, or replace the use of animals in research and testing" (NTP 1999, 16). 
NTP’s Advisory Committee on Alternative Toxicological Methods oversees the programs 
of NICEATM and ICCVAM and advises them on activities and priorities. The Advisory 
Committee is "composed of knowledgeable representatives from academia, industry, 
public interest and animal welfare organizations, other agencies, and the international 
community" (NTP 1999, 17), and its members are selected in much the same way as 
members of BSC. When ICCVAM evaluates a proposed alternative method, it convenes 
a peer review panel to develop "a scientific consensus on the usefulness, limitations, and 
validation status” (NTP 1999, 17), the conclusions of which are then forwarded to 
regulatory agencies for judgments according to their unique mandates. 

Office of Medical Applications of Research (OMAR) 

OMAR, part of the National Institutes of Health, runs consensus development 
conferences for the purpose of assessing and transferring new medical knowledge and 
technology to medical practitioners. Created to manage a controversy over breast cancer 
screening in 1977, consensus development conferences became the primary mechanism 
for technology transfer at NIH through the early 1980s (OTA 1982), and OMAR has 
disseminated more than 120 consensus and technology assessment statements produced 
through this process. 26 

Consensus statements are written by "broad-based, independent panels of non-Federal, 
nonadvocate individuals knowledgeable in the field of medical science under 
consideration." 27 Panels typically include medical researchers, providers, 


25 Along with three other internal committees — the RG1, RG2, and the Executive Committee, which are all 
composed of Federal employees from NTP and its participating agencies — the Subcommittee votes on 
each substance nominated. Based on the votes of all the committees, the director of NTP makes 
recommendations to the Secretary, DHHS, who issues the Report on Carcinogens and its list of substances 
known and reasonably anticipated to be human carcinogens. Substances nominated for delisting are 
handled the same way. 

26 See httpy/odp.oilnih.eov/consena.is/about/abouLhtm 

27 Denmark altered the consensus conference format by replacing the expert panel with lay citizens. Such 
citizens' panels have become popular ways in Europe and elsewhere of conducting technology assessments 
and enhancing public understanding of science. 
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methodologists, and a public representative; to avoid the appearance of conflict, panelists 
may not be Federal employees. Panels range in size from nine to sixteen, but average 
twelve or thirteen. 

The panel hears testimony from experts and the public for one and one-half days and then 
convenes in executive session to draft the statement which, upon completion on the third 
day, is circulated among the broader audience for comment. Because the conference "is 
an exercise in peer review, in which the panel reviews information presented by the 
experts, panelists should not be speakers." The panel then revises the statement, which is 
later released (on WWW and in the lay press) and usually published in the professional 
medical literature. According to OMAR, the panel and any recommendations are not 
advisory to NIH and "[a] consensus statement is not a policy statement of NIH or the 
Federal Government." 

Topics for consensus development conferences are generated from recommendations 
across government and the public.' 18 After a topic is chosen, a conference takes nine 
months to one year to schedule plan, as a planning committee — composed of government 
employees with particular expertise, who have been screened for conflicts — meets to 
select panelists, presenters, and articulate the specific questions that structure the 
conference. 

Some commentators (Markle and Chubin 1987; fOM 1990) have recommended that 
OMAR expand the purview of consensus development conferences beyond narrow 
technical considerations. To some extent, this expansion has occurred, but "the primary 
purpose of a CDC is to provide scientific evaluation. To this end, the statements in 
general and the recommendations issued in particular should focus on medical safety and 
efficacy, although they may refer to other issues (e.g., economic, sociologic, legal, and 
ethical] in passing." 

National Marine Fisheries Service (NMFS, or NOAA Fisheries) 

NMFS has a mandate to provide the "best science available" to regional councils and 
others who participate in developing fisheries management plans. Peer review, using 
both internal and external reviewers, is an important part of that process. 

Two mechanisms of internal review are stock assessment workshops (SAWs) and stock 
assessment research committees (SARCs), which review stock assessments for fish 
species under management. Workshops and committees are conducted at regional 
science centers and deal with fish species in those regions. SAWs and SARCs vary in 
depth and duration — as short as an afternoon or as long as a week - depending on how 
much advance preparation is required. Staff scientists, who develop the stock 
assessments, attend the meetings. Occasionally, an external speaker may attend to 
discuss new developments. SAWs are open to the public. 


28 Consensus development conferences synthesize new information, and thus they differ from the clinical 
guidelines offered by the Agency for Healthcare Research and Quality, which synthesize information about 
existing practice. 
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The stock assessments derived from SAWs and SARCs are submitted to the relevant 
Regional Fishery Management Council for that species, along with recommendations for 
the development of the management plan. The council's Science and Statistical 
Committee then further reviews the stock assessment. SSCs are composed of biologists, 
statisticians, economists, and social scientists from academia and state natural resource 
agencies. SSCs do not have to accept the recommendations of NMFS scientists to 
develop the management plan. They may also question the conclusions of the stock 
assessment when making their recommendations to the council. NMFS may have stock 
assessments externally reviewed if there is a problem achieving the requisite consensus. 

NMFS uses internal and external review for other science used for regulatory purposes. 
Regional councils submit data that may be used to designate ocean or coastal 
environments as Essential Fish Habitats. The underlying science, as well as science 
supporting any potential restoration of habitat, is reviewed externally and internally. All 
review is done in an open fashion by both panels and individuals, and instructions given 
to reviewers are specific to the product being reviewed- NMFS identifies external 
reviewers through institutions such as the National Research Council, professional 
organizations (e.g., American Fisheries Society), recommendations of other scientists, 
and academia. 

The Northeast Regional SAW conducts peer review of stock assessments twice each 
year. 29 The SAW determines which species to assess and establishes the terms of 
reference for the assessment. Each SAW cycle begins and ends with the meeting of the 
SAW Steering Committee and includes: Working Group meetings, where analyses are 
prepared for review; a five-day SARC meeting, where analyses are peer-reviewed and 
advice for managers is developed; and a two-session Public Review Workshop. 
Participants in this process include: NMFS scientists and managers; representatives from 
state fishery agencies, regional fishery management councils and fishery agencies outside 
the region; academic and other non-governmental participants; and industry members. 30 
The time frame for assessments is generally dependent on legislatively mandated 
deadlines. 

NMFS has a Division of Research, Analysis and Coordination (ST2) within its Office of 
Science and Technology. ST2 is responsible for implementing NMFS's Strategic Plan for 
Fisheries Research, as well as for helping to ensure "the quality and credibility of NMFS 
fisheries science." 31 ST2 supports the NMFS's decisions on national standards for 
fisheries management. Among ST2's major projects in science quality assurance are: 


25 The SAW Steering Committee, composed of directors of NEFSC and NERO, the Mid-Atlantic Fishery 
Management Council [MAFMC], New England Fishery Management Council [NEFMC], and the Atlantic 
States Marine Fisheries Commission [ASMFC]), directs the process. 

30 The Sustainable Fisheries Act has also mandated NRC review of stock assessments for important species. 

31 See http^Avww.stnmfs.gov/st2/review5.html . 
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• NMFS Science Center Accreditation and Certification, which will use an independent 
panel of experts to set benchmarks for the quality and integrity of Fisheries science 
data; 

• Center of Independent Reviewers, which will create a pool of external experts whom 
the agency tap to review the stock assessments, new management methods, scientific 
programs, and grant proposals, and to participate in the design of those enterprises. 32 

• Methodological Peer Reviews, conducted in coordination with NMFS science 
centers, to produce reliable data for the regional fishery management plans, e.g., the 
Northeast Fisheries Science Center's Workshop (SAW—16) on the American lobster 
fishery. 

• National Peer Reviews, mandated by the Magnuson-Stevens Fishery Conservation 
and Management Act (Magnuson-Stevens Act), the scope of which ranges from 
developing recommendations for applying ecosystem principles in fishery 
conservation to evaluating the research and management of red snapper in the Gulf of 
Mexico. 

• NMFS Stock Assessment Improvement Plan, which will implement ten 
recommendations contained in a report. Improving Fish Stock Assessments, by the 
National Research Council's review of the current stock assessment methods and 
models. 

National Academy of Sciences 

Initially chartered by Congress in 1863, the National Academy of Sciences — and the 

associated Institute of Medicine and the National Academy of Engineering ~ is an 
honorific and an advice-giving body. The operating arm of the National Academy 
complex is the National Research Council. The Academy complex conducts studies from 
requests by Congress, 33 federal agencies, and foundations, and is subject to only partial 
public disclosures under the Federal Advisory Committee Act Amendments of 1997 (P.L. 
105-153). In this role, the Academy complex serves a function akin to peer review for 
federal agencies, often evaluating programs, helping to set long-term research priorities, 
and providing other scientific and policy advice. The Academy complex identifies risk- 
related reports as "among the most difficult and potentially controversial" of its activities 
(NRC 1998). 

Staff of the Academy complex choose members for its committees, with the advice of 
standing boards and commissions and the final approval of the chair of the NRC (who is 
also president of the NAS; NRC n.d.). The Academy complex recognizes that the 
"credibility of a report can be weakened if the committee that produced it is perceived to 
be biased" (NRC 1998:9), and to this end it requires committee members to complete a 
list of "potential sources of bias and conflicts] of interest" and discuss it with their fellow 
members. Only on rare occasions are potential biases or conflicts publicly disclosed or 
do they serve to disqualify a potential member. The Academy complex calls for 


32 Congress recently appropriated Hinds for the compensation of external reviewers who participate in the 
Center of Independent Experts 

33 Congress does not appropriate directly to the Academy complex, however, but funds it through agencies. 
A small number of studies are self-initiated. 
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committee members to "separate fact from opinion, analysis from advocacy" (NRC 
1998:12), but nevertheless recognizes that in "studies involving incomplete or 
inconclusive data, the collective experience and reasoned judgment of committee 
members become crucial factors in developing conclusions and recommendations" (NRC 
1998:13). 

The Executive Committee of the NRC's Governing Board reviews and approves study 
proposals. All studies, as well as workshop proceedings and other documents, undergo 
"an independent review by anonymous experts who were not involved in the report's 
preparation" (NRC 1998:17). The Academy believes that this "independent, rigorous 
review is a hallmark that distinguishes the NRC/IOM from many other organizations 
offering scientific and technical advice on issues of national importance" (NRC 1997). 
The Report Review Committee, composed of about 30 members of NAS, NAE, or IOM, 
oversees the review, which is managed by the board or commission in charge of the 
study. RRC ensures that: "the report addresses the approved study charge and does not 
go beyond it; the findings are supported by the evidence and arguments presented; the 
exposition and organization are effective; and the tone of the report is impartial and 
sensitive policy issues are treated with appropriate care" (NRC 1998:18). Study 
committees have an opportunity (and are required) to respond in writing to report 
reviews. A monitor, appointed by RRC (and often a member of it), and/or a review 
coordinator, appointed by the commission or board, evaluates the responses to the review 
and defends the interests of the reviewers. NRC does not release reports until the 
monitor, the coordinator, the sponsoring board or commission, and the study panel are all 
satisfied. Names of principal reviewers are released after the report is published, but 
reviews themselves are single-blinded. This review by RRC is highly formalized and 
may include the services of 20 or more reviewers and produce a review record longer 
than the report itself. 

States 

There is no systematic information about the use of RPR in the states, although there is 
evidence of real interest among the states in the role of science in policy making 
(Andersen 2000; CGS 1999). According to a source at the National Conference of State 
Legislatures, a vast majority of states rely on US EPA for their scientific and technical 
analyses pertaining to environment. 

One exception is California, where peer review is a robust part of the regulatory process. 
In 1997, Governor Pete Wilson signed SB 1320, which contained a peer review 
mandate. 34 Also in 1997, the Risk Assessment Advisory Committee issued a final report 
that asked for peer review to be applied more consistently throughout Cal/EPA, The 
1998 Strategic Plan seeks to "[ejnhance the use of external peer review of scientific rules 
to ensure that the best science is used in all of Cal/EPA's environmental efforts," and that 


34 California's Proposition 65 also mandates a peer function: "A chemical is listed if the 'stale’s qualified 
experts'—an independent panel of scientists and health professionals appointed by the Governor—find tiiat 
the chemical has been clearly shown to cause cancer or birth defects or other reproductive harm." See 
www.calepa.ca.cov/miblications / f3ctsheets/1997/prop65fe.htm. 
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yearCal/EPA (1998) published a "Policy and Guiding Principles for External Scientific 
Peer Review." 

Parts of Cal/EPA use peer review for different regulatory purposes. The 1997 legislation 
specified external peer review for particular work products of Cal/EPA's several 
divisions. For example, the Air Resources Board must review its ambient air quality 
standards and its exposure assessments for toxic air contaminants (Cal/EPA 1997). ARB 
describes extensive opportunities for peer review in its pre-regulatory "Work Plan for 
Conducting an Airborne Environmental Fate and Transport Analysis of Ethanol- 
Containing Gasoline." 35 The Department of Pesticide Regulation sends draft risk 
assessments out for peer review (for 24 active pesticide ingredients in the year prior to 
mid-1999), Cal/EPA (I998:App.B) recommends a different type of peer review panel 
depending on the nature of the issue and the expected product. For example, it uses a 
Scientific Review Panel to review risk assessments for airborne toxics prior to 
rulemaking, but it uses the National Academy of Sciences for reviewing comprehensive 
risk assessments, e.g., of methyl bromide. SB 1320 requires review for "any rule, as 
defined, establishing a regulatory level, standard, or other requirement for the protection 
of public health or the environment," but Cal/EPA recommends consideration of reviews 
for many internal, preliminary, or other work products or analytical methods. 

Cal/EPA (1998:4) views peer review as "an objective, critical review of a draft Agency 
scientific work product typically by independent scientific experts" that can occur at 
"several discrete points...[but] is typically characterized by a one-time interaction or a 
limited number of interactions... [and] can also occur during the early stages of a project 
or methods selection." It intends peer review to enhance and improve the agency's 
technical work, rather than to serve as mere legitimation for the agency. Cal/EPA 
distinguishes between formal and informal peer review, as the former can more easily 
"ensure the transparency of agency regulatory actions and enhance the credibility of 
agency decisions" (1998:4), and between peer review and public comment, as the former 
usually occurs prior to public comment and the latter is open to issues beyond the 
scientific purview (1998:5). Cai/EPA holds that peer reviewers should be independent, 
meaning that they should not have participated directly or indirectly in the development 
of the product being reviewed, and that they should not be employed by the agency 
(advisory committee members are not considered employees). 

There are a few particular, and possibly even idiosyncratic, uses of peer review in support 
of policy or regulatory decision making in states: 

• The Michigan Environmental Science Board — established by Governor John Engler 
by executive order in 1992 - is "an independent autonomous state agency established 
to provide sound scientific and technical advice to the Governor of Michigan and to 
state departments, as requested by the Governor, on matters affecting the protection 
and management of Michigan's environment and natural resources."^’ The Board 
answers referrals from the Governor, who asks specific questions. A subcommittee 


35 See www.caleDa.ca.gQv/programs/mtbe/outline/carfa.pp t. 

36 See wwvv.mesb.org . 
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of the Board, with guests added as needed for their expertise, then provide answers. 
Referrals may include the review of: environmental impact statements, proposed 
environmental standards for permits or operating licenses, the methodology for 
establishing permit or operating license conditions that contain environmental 
standards that have not been established by administrative rule, and staff actions on 
permits and licenses. The Board's meetings are open, but there is no formal public 
comment. 

• North Carolina has a standing Scientific Advisory Board on Toxic Air Pollutants, 
composed of five scientists who review new or revised acceptable ambient level 
guidelines for air toxics and who conduct risk assessments. Meetings of SAB are 
open to the public "and participation by interested groups is not only allowed, but 
encouraged." 37 A broader Environmental Management Commission adds economic 
and feasibility concerns, and an Environmental Review Commission serves as 
gatekeeper to the legislative process. 

Defining Peer Review 

There is no authoritatively fixed definition of peer review in any arena — funding, 
journal, or regulatory. Appendix 1 shows the diversity of definitions of FPR that federal 
funding agencies use. Nevertheless, GAO's observation that FPR across the government 
involves the review of scientific or technical merit by competent persons with no 
unresolved conflicts of interest seems to encompass the variety of practice — not just for 
FPR but also for JPR and RPR — as long as the purpose of the review is explicitly added. 

Key to this definition's utility, however, is further defining what constitutes the difficult 
concepts of "scientific or technical merit," "competent persons," and "no unresolved 
conflicts." Indeed, in seemingly all circumstances of FPR and JPR, yet alone RPR, 
scientific or technical merit is supplemented by some concern for mission or audience 
relevance, programmatic or editorial balance, economic benefits, and other less strictly 

scientific or technical concerns. Often, the practice of peer review defines "competent 
persons" as including criteria in addition to strictly technical considerations. Managers of 
review may actively seek reviews from an intentionally diversified group of peers; the 
striking exception is "serial review" in the physical sciences, where an editor may make a 
determination based on the advice of a single reviewer. 

The treatment of conflicts, as well as what it means to resolve them, is similarly diverse. 
In some instances, e.g., medical journals, consulting, membership, and employment 
relationships in and of themselves are sufficient for disqualification from publishing 
certain types of articles. In other circumstances, financial relationships, once disclosed, 
are permissible. Some funding agencies consider not just financial but also professional 
relationships and affiliations as potentially disqualifying, although they often rely on the 
judgment of potential reviewers and program managers to resolve such conflicts. 

Although peer review may be a social practice that is a sine qua non of contemporary 
science, there is no one formula or recipe for its practice apart from its specific contexts. 

37 See daq.state.nc.us//officesfechricalfoxics/nsk/ . 
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Lessons from Funding and Journal Peer Review 

Nevertheless, there are likely some lessons from the practice of FPR and JPR that may be 
relevant for the practice of RPR. The diversity of practice, representing differing 
histories, disciplines, visions, and so forth itself suggests that a diversity of practice in 
RPR may be expected and, perhaps, appropriate. 

Despite its diversity of practice, the jurisdiction of FPR and JPR is procedurally clear 
and encompassing, but subject to modest discretion. These characteristics could be goals 
for RPR. 

The format of FPR and JPR is flexible within bounds, and decision makers retain a great 
deal of discretion in the face of peer advice. Agencies that use advisory committees for 
RPR are likely familiar with this flexibility and discretion, although public, legislative, 
and judicial scrutiny complicates the notion of agency discretion. Achieving the same 
balance of flexibility and regularity in RPR as JPR and FPR accomplish may be difficult 
because of the strong and diverse interests and the legalized nature of the regulatory 
process. 

FPR and JPR may take in the upwards of eight months to complete, and the material 
being reviewed is, in such cases, usually limited to 20 or 25 pages in length and to a 
relatively narrow field of practice or set of techniques. RPR would likely encompass a 

scope of materials an order of magnitude larger and more diverse and interdisciplinary in 
character. Such scope usually does not pose problems for committees but could pose 
problems for individual, ad hoc reviewers. 

Instructions given to funding and journal peer reviewers are general and include 
considerations, including those mentioned above like programmatic, administrative, or 
editorial criteria, beyond strictly scientific merit. EPA, at least, recognizes that charges to 
peer reviewers are critical. But RPR nevertheless seems to use similarly vague 
instructions — thus conflicts between agencies and advisory committees over what 
constitutes substantial evidence or adverse health effects that the literature documents. 
Furthermore, to the extent that regulatory peer reviewers extend their evaluations beyond 
strictly scientific merit, the less such reviews serve agency goals for either quality 
validation or legitimacy. The legislative proposals for RPR give little guidance for how 
potential reviewers are to be instructed. 

The participation in FPR and JPR is subject to great discretion by the program manager 
or editor, but substantive expertise and identifying and resolving conflicts of interest are 
critical. Program managers and editors are expected to exercise intellectual leadership, 
helping define what it means to be part of the field in which they are funding or 
publishing research. This leadership clearly influences the choice of reviewers and the 
treatment of reviews (and researchers' responses to them). Agency officials are expected 
to exercise some political discretion, based on values held by their elected principals and 
the public, and these political considerations will clearly influence the choice of 
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reviewers and the treatment of reviews. It is not clear how much peer review — if it 
remains advice and if outcomes are significantly dependent upon the choice of reviewers 
— itself can manage to constrain discretion beyond some modest addition of quality. 

Both FPR and JPR are well-institutionalized in the social sciences, but there are some 
differences between review in the social and natural sciences that suggest that there is 
less of a consensus among practitioners of the former about what constitutes high quality. 
This finding seems to carry conflicting lessons for RPR, as there is no reason from 
practice not to subject regulatory social science to review, but that the application of such 
review may be more likely to result in divergence of opinion (or pure legitimation) rather 
than refinement of opinion. 
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Synthesis on Regulatory Peer Review 

Regulatory and other evaluative and assessment agencies appear to have made significant 
strides over the last decade in reforming and improving the way that they incorporate 
science into decision making. Impetus for this improvement has come from concerns for 
sounds science, as well as from strategic planning efforts and, perhaps, from constituency 
building efforts. Examples include EPA's Science Policy Council and its Peer Review 
Handbook , FDA’s incorporation of peer review goals and methods into its strategic 
planning, and the National Toxicology Program's creation of its Board of Scientific 
Counselors. 

The jurisdiction of RPR is more variable and flexible than FPR and JPR, except in cases 
of legislative mandates or engaged agency effort. The jurisdictions of legislatively 
mandated advisory committees, for example, is relatively clear, as is the jurisdiction of 
their agency-created counterparts. But unless an agency undertakes the effort that EPA 
put into its Handbook, it is not clear that the appropriate variety of work products — 
including models, analytical and statistical methods, and other agency research — will be 
peer reviewed. It is not clear at EPA if a thorough evaluation of the implementation of 
peer review procedures has occurred. 

Almost all peer review — whether funding, journal, or regulatory -- is responsive in its 
jurisdiction; that is, it reviews what is brought to it, but it does not actively seek out 
material to review. FDA's auditing process is somewhat different, as it reviews the 
conditions for, and the production of, new drug and other data in a random and for-cause 
basis. Given controversies over the access to data that forms the basis of regulatory 
decisions, some commentators (Cohen and Hahn 1998) have called for a more active 
public role in assuring data integrity'. 

The format of RPR usually involves FACA committees, although there are a number of 
other models including government-only panels (which are often coordinating as much as 
review committees) and individual, ad hoc review. FACA committees are technically 
advisory, but decision makers ignore them at real hazard of public, congressional, or 
judicial scrutiny. Accounts of the work of FACA committees in RPR (e.g., JasanofF 
1990) suggest that a flexible and ongoing negotiation between the committee and the 
agency, begun early in the work product, will be most successful. 

The instructions involved in RPR vary in details, especially because RPR is applied to a 
variety ot work products. Committees performing RPR may or may not have specific 
guidelines for their analyses, and there can be conflicts between committees and agencies 
over details that could have been more clearly specified, e.g., how to determine an 
adverse health risk. In instances without clear instructions, agencies engaging in RPR 
allowing reviewers an enormous amount of discretion in coming to influential judgments. 
EPA's Handbook recognizes this difficulty in its emphasis on crafting a high-quality 
charge to reviewers - as does the National Academy complex. 
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Some RPR. specifies that only peer reviewed and publicly available documents should be 
used as inputs to decision making (e g., NTP's Report on Carcinogens), other RPR is less 
strict (e.g., FDA's new rule for off-label uses includes non-peer reviewed reference 
journals, and agencies with public meetings cannot help but hear material that was not 
peer reviewed, although they may disregard it). The Supreme Court's Daubert decision 
held that peer review should be important but not dispositive in a trial judge's decision to 
include expert testimony. Some agencies (again, NTP is a good example) also have clear 
instructions for the threshold of decision making, for example, the number and type of 
studies of a particular finding that lead to a judgment of "known human carcinogen." 
Although this research could not be detailed enough to investigate those kind of 
instructions for each agency (or for each committee within each agency), the literature 
suggests that such clarity is not common but that it may be crucial to outcomes. 

Participation is largely governed by FACA for the formal advisory committees, and thus 
some form of disclosure of conflicts, balance of membership, and public openness is 
standard. Non-FACA committees are often narrowly technical or broadly inter-agency. 
Even advisory bodies exempt from FACA — the National Academy complex and NAPA 
— have at least some recognition of role of balance and public openness, although it is 
much less robust under these circumstances. EPA's Handbook suggests that the 
solicitation of individual as opposed to consensus opinion is not subject to FACA, and 
thus participation in RPR could be more easily at the discretion of agency managers, but 
EPA also provides for some oversight of RPR as well. EPA also attempts to distinguish 
between simple peer involvement and more dedicated peer review, and between public 
review (which is not based in expertise and happens toward the end of a process) and 
peer review (which is based in expertise and happens early in a process). 

Synthesis Reviewing Proposed Legislation 

There are several legislative alternatives represented by bills introduced in the 106 th 
Congress promoting RPR; the major bill involves a comprehensive effort across all 
regulatory agencies, while other minor bills involve single agencies. Most comments will 
pertain to the major bill, S. 746. Appendix II contains relevant text from these bills, and 
Appendix III contains relevant excerpts from Senate Report 106-110, accompanying S. 
746. 

The proposals are generally clear about the jurisdiction of the RPR sought, which would 
be based on the estimated fiscal impact of the proposed regulation. This jurisdiction 
expresses an understanding that there are real costs to performing RPR, as EPA has 
acknowledged in its Handbook. The legislation would also assure that cost-benefit 
analyses (in effect, a social science input to regulatory decisions) would be peer reviewed 
as well. It would not address the wide variety of other work products, however. If it is 
the case, as Zuckerman and Merton (1971) speculate, that there is less consensus in the 
social sciences about what constitutes appropriate work, then it is possible that RPR of 
social science and economic analysis will not be convergent but will result in controversy 

and delay. 
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The format for RPR described by the legislation would be flexible, allowing either 
committee or ad hoc review. There would be strict time limits on the review and agency 
officials would be required to respond to concerns raised by peer reviewers, although the 
precise relationship between the review, the response, and the details of the regulation at 
issue is not clear. 38 The review would occur prior to public notice. There is also some 
uncertainty about the nature of anonymity for ad hoc reviewers and the publicity of the 
reviews. The magnitude of the material submitted for RPR may be significantly greater 
than that submitted for FPR or JPR, and thus the time required to review it may be even 
longer than the variable but usually months-long process for grants and manuscripts. 

The proposals are vague about what kind of instructions would be given to referees, of 
particular concern for ad hoc reviewers. Although the proposals specify a process for 
agencies to follow' after RPR, it is somewhat formalistic and, like the Administrative 
Procedures Act and the process for incorporating public input, does not completely 
specify how significant negative or positive reviews should be for the agency's final 
decision making. 

The proposals would exempt the described RPR from FACA, replacing it with 
participation from referees whom the legislation presumes could be unbiased (although 
the legislation anticipates that standing FACA committees could also be used, but not in 
FACA mode). Agencies would determine policies for conflicts. It is possible that 
agencies would use the legislation to end-run around existing FACA committees if such 
committees are viewed as roadblocks to regulatory proposals. 


38 This relationship may be something like that in the Administrative Procedures Act, which requires 
agencies to respond to public comments submitted in response to proposed rules, but does not require 
agencies to alter the proposed rules. 
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